The Meanings of Objects and Actions 1 Running head: THE MEANINGS OF OBJECTS AND ACTIONS Representing the Meanings of Object and Action Words: The Featural and Unitary Semantic Space (FUSS) Hypothesis

This paper presents the Featural and Unitary Semantic Space (FUSS) hypothesis of the meaning of object and action words. The hypothesis, implemented in a statistical model, is based on the following assumptions: First, it is assumed that the meanings of words are grounded in conceptual featural representations, some of which are organized according to modality. Second, it is assumed that conceptual featural representations are bound into lexico-semantic representations that provide an interface between conceptual knowledge and other linguistic information (syntax and phonology). Finally, the FUSS model employs the same principles and tools for objects and actions, modeling both domains in a single semantic space. We assess the plausibility of the model by showing that it can capture generalizations presented in the literature, in particular those related to category-related deficits, and show that it can predict semantic effects in behavioral experiments for object and action words better than other models such as Latent Semantic Analysis (Landauer & Dumais, 1997) and similarity metrics derived from Wordnet (Miller and Fellbaum, 1991). The Meanings of Objects and Actions 3 Representing the Meaning of Object and Action Words: The Featural and Unitary Semantic Space (FUSS) Hypothesis "To grasp the meaning of a thing, an event, or a situation is to see it in its relations to other things; to note how it operates or functions, what consequences follow from it, what causes it, what uses it can be put to. In contrast, what we have called the brute thing, the thing without meaning to us, is something whose relations are not grasped" (John Dewey, 1910; quoted in Miller and Johnson-Laird, 1975, p. 212) In the above passage Dewey simply and clearly points to two fundamental questions in the investigation of meaning. First, how are meanings related? Second, how is each meaning structured? These two separate but highly related questions have traditionally been central in cognitive psychology, linguistics, psycholinguistics and neuroscience. With respect to meaning relations, the intuitive impression that some meanings are more similar than others is compelling. The role of similarity in meaning is well documented in a variety of behavioral tasks and has a major role in theories of semantic representation developed within linguistics, cognitive psychology and neuroscience. In the linguistic literature, for example, some researchers argue that words are organized into semantic fields: content domains (in which different elements are lexicalized in a language) structured on the basis of fine-grained contrasts and properties of similarity among their exemplars that govern the relations among the different words. These principles are relational (e.g., contrast) and the meanings of words within and between fields are defined solely with reference to other words (Kittay, 1987). Meaning similarity also plays a major theoretical role in a number of psychological theories of semantic memory (e.g., Collins & Quillian, 1969; Collins & Loftus, 1975; Medin, Goldstone & Gentner, 1993; Miller & Johnson-Laird, 1975; Tversky, 1977). Effects of meaning similarity are well-documented in linguistic tasks. For example, in spontaneous speech, mistakenly substituting one word with another word similar in meaning is one of the most common types of slip of the tongue (Garrett, 1992a). In naming experiments, the presentation of a word similar in meaning to a target picture immediately prior to the target has interference effects (e.g., The Meanings of Objects and Actions 4 Schriefers, Meyer and Levelt, 1990). In word recognition experiments, semantically related primes speed up target recognition (McRae & Boisvert, 1998). The fact that more similar entities are clustered together and are separate from other less similar entities is also evident in the neuropsychological literature: semantic knowledge can break down selectively along category boundaries, broad boundaries such as selective deficit (or sparing) of living and non-living things; but also more fine-grained distinctions such as specific impairments (or sparing) of body-parts; fruits and vegetables among others (see Forde & Humphreys, 2002 for updated reviews). With respect to the issue of how each meaning is represented, central to a number of theories of semantic organization is some sort of featural decomposition (see, e.g., Collins & Quillian, 1969; Jackendoff, 1990; Minsky, 1975; Norman & Rumelhart, 1975; Saffran & Sholl, 1999; Shallice, 1993; Smith & Medin, 1981, for overviews). Properties of featural representations such as correlations among features, features shared among different exemplars, and features distinctive of given exemplars have been used to account for meaning similarity effects (Malt and Smith, 1984; McRae, de Sa & Seidenberg, 1997; Smith, Osherson, Rips and Keane, 1988; Smith, Shoben and Rips, 1974). Featural correlations and feature types (i.e., whether features refer to visual, acoustic, motoric, etc. properties of a concept) have also been used to explain category-related impairments of semantic knowledge (Caramazza, Hillis, Rapp & Romani, 1990; Devlin, Gonnerman, Andersen & Seidenberg, 1998; Tyler, Moss, Durrant-Peatfield & Levy, 2000; Warrington & Shallice, 1984). Whether treated in terms of featural systems or other mechanisms, these regularities of normal and impaired meaning processing provide a major part of the empirical challenge to the development and elaboration of psychological theories of meaning representation. In this paper, we present the Featural and Unitary Semantic Space (FUSS) hypothesis of the representations of words referring to objects and actions. FUSS integrates a number of previous claims and research strands encompassing neuroscience, cognitive psychology and psycholinguistics into a working hypothesis of how the meanings of words are represented, which we have operationalized into a statistical model. Word meanings must be grounded in conceptual knowledge; however, at the same time the two levels cannot be equated (Murphy, 2002). For this reason we explicitly specify the type of conceptual representations we assume to exist, and we also explicitly specify the manner in which The Meanings of Objects and Actions 5 conceptual representations are linked to the meanings of words. Moreover, our hypothesis is constrained by neuropsychological and neuroanatomical evidence, in addition to having psychological plausibility. Given these premises, FUSS is formulated based on the following assumptions: First, featural representations are essential components of conceptual structure, and at least some of these featural representations (those for concrete concepts) can be described as modality-specific. Second, these featural representations are bound into lexico-semantic representations whose organization is dictated by featural properties (such as shared and correlated features). These first two assumptions lead us to a third assumption, namely, that the same general computational principles can be used to encode the representations for words in different content domains (here: objects and actions). Thus, in FUSS, the semantic system, organized according to the same principles for the object and action domains, comprises two levels of representation: conceptual structures (of which modality-specific features are an essential part) and lexico-semantic representations (derived from the conceptual features and organized on the basis of featural properties). Each of these architectural assumptions has existing exemplars in cognitive psychology, linguistics or neuroscience. Very few attempts, however, have been made to integrate them (a notable exception being Jackendoff, 2002). In contrast to other proposals, we integrate these assumptions across domains of object and action knowledge into a quantitative model based on feature norms. The paper is organized into two main sections. In the first section, we discuss the underlying assumptions, motivating them on the basis of previous work. We present an implementation based on these assumptions and we detail analyses of the properties of the implemented model that relate back to the assumptions. In the second section, we present a series of behavioral experiments that test the ability of the implemented model to predict semantic similarity effects for both objects and actions. In this section, we directly compare the predictive power of our hypothesis to the predictive power of two existing models: Latent Semantic Analysis (LSA, Landauer & Dumais, 1997) and similarity measures derived from Wordnet (Miller & Fellbaum, 1991), models that are based on different principles but both of which allow us to derive quantitative predictions for both objects and actions. The Featural and Unitary Semantic Space (FUSS) Hypothesis: The Meanings of Objects and Actions 6 Assumptions and Implementation Conceptual Representations: Features and Feature-Types Featural conceptual representations are assumed by a number of authors (see, e.g., Collins & Quillian, 1969; Jackendoff, 1990, 2002; Minsky, 1975; Norman & Rumelhart, 1975; Saffran & Sholl, 1999; Shallice, 1993; Smith & Medin, 1981), although few researchers have attempted to assess whether conceptual structures can be exhaustively decomposed into a set of primitive features (see Fodor, Fodor & Garrett, 1975; Fodor, Garrett, Walker & Parkes, 1980; Jackendoff, 1983, 2002) and featural views have been criticized because they do not take into account inferential knowledge (Gopnick & Meltzoff, 1997; Medin, 1989; Murphy and Medin, 1985). In order to underscore the representations of multitudes of concepts from concrete to abstract, from referential to relational, such featural representations must encompass a variety of types. However, at least for those concepts referring to concrete entities and activities in the world they must be grounded in our interactions with the environment, namely, perception and action (see e.g., Allport, 1985; Barsalou, 1993; Damasio, Grabowski, Tranel, Hichwa & Damasio, 1996; Glenberg & Robertson, 2000). Thus, a further related assumption is that distributed features are represented according to the modality by which we acquire and experience events and things (i.e., a modality-specific manner). If conceptual organization is expressed in terms of featural representations, then specific features can be selectively activated, and their effect can be observed in automatic tasks. Behavioral studies have shown that the association of a given feature with a given word is sensitive to contextual information. Tabossi (1988) showed that the activation of given features (e.g., or for "lemon") can be enhanced by context (see also Barsalou, 1982;Tabossi & Johnson-Laird, 1980). Reaction time studies (using lexical decision or word naming tasks) which have attempted to establish whether specific features can be automatically activated have provided somewhat mixed results: while some authors have shown priming effects for word pairs such as "button coin" (Moss, Ostrin, Tyler & Marslen-Wilson, 1995; Schreuder, Flores d'Arcais & Glazenborg, 1984), at least for some tasks and SOAs (Stimulus Onset Asynchronies) between prime and target, other studies failed to observe priming for perceptual features (Pecher, Zeelenberg & Raaijmakers, 1998). Crucially, however, Kellenbach, Wijers and Mulder (2000) The Meanings of Objects and Actions 7 have shown that response times in tasks like lexical decision may not be a sensitive enough measure. In fact, they observed robust priming for perceptually related pairs as a modulation of the N400 component of ERPs (a component associated with semantic processing) while no effect was observed in the response times. The relevance of featural representations across categories is also revealed in spontaneously occurring speech errors. Substitution errors such as saying "wheel" when "foot" is intended (Garrett, 1992a) suggest that shared features related to motion can be sufficiently active to induce an error in which, importantly, semantic field (category) membership is not preserved. This observation in production may be considered the counterpart of the context effects observed in comprehension briefly reviewed above. In the neuropsychological literature, modality-specific featural representations are postulated by a number of authors (Allport, 1985; Farah and McClelland, 1991; Martin & Chao, 2001; Saffran, 2000; Warrington and McCarthy, 1987; Warrington & Shallice, 1984). However, results are mixed with respect to the question of whether specific types of features can be selectively impaired by brain damage. Some patients have been shown to have both category-related deficits for living things and also deficits in the appreciation of perceptual features of concepts, especially in the living things domain (e.g., Basso & Capitani, 1988; De Renzi & Lucchelli, 1994; Farah, Hammond, Mehta & Ratcliff, 1989; Forde, Francis, Riddoch, Rumiati and Humphreys, 1997; Gainotti & Silveri, 1996; Marshall, Pring, Chiat & Robson, 1996a; Rosazza, Imbornone, Zorzi, Farina, Chiavari & Cappa, 2002; Sartori and Job, 1988; Silveri and Gainotti, 1988). Additionally, in some patients the category-related deficit does not strictly respect the boundary between living and non-living things but encompasses exemplars from various fields arguably related on the basis of the weight of perceptual or functional properties. For example, patient JBR reported by Warrington and Shallice (1984) had problems with concepts referring to living things but also musical instruments and gemstones. In contrast, patient YOT had problems with artifacts but also body parts (Warrington & McCarthy, 1987). Warrington and colleagues argued that musical instruments and gemstones are similar to living things in that they are distinguishable in terms of perceptual features, whereas body parts and artifacts are categories of knowledge for which function is salient. Interestingly, some studies have shown that impairments in the appreciation of perceptual features are also correlated The Meanings of Objects and Actions 8 with the patients' ability to name and distinguish actions which are differentiable on the basis of their manner (e.g., "slide" and "crawl") which can be considered a perceptual distinction (Marshall, et al., 1996a; Marshall, Chiat, Robson & Pring, 1996b). Perceptual knowledge and knowledge of living things are not necessarily conjointly impaired in pathology, however (e.g., Caramazza & Shelton, 1998; Laiacona, Barbarotto & Capitani, 1993), leading to the development of alternative theories that either assume a featural basis of conceptual knowledge without entailing the segregation of knowledge into modality-specific systems (e.g., Caramazza et al., 1990; Tyler et al., 2000), or assume a genetically based commitment of neural tissue for the recognition of animals and plants (Caramazza, 1998). A crucial consideration is that, most of the findings that have been described as damaging for modality-specific views of featural representations are damaging only under the (parsimonious but not necessary) assumption that category-related deficits arise as a consequence of a single type of impairment (see Vinson, Vigliocco, Cappa & Siri, 2003 for a discussion). This fact is elegantly demonstrated for object concepts by the comprehensive analyses by Cree and McRae (2003) who show that although feature types alone cannot account for all of the main trends in the patients’ data, they are an important component. A modality-specific organization is compatible with anatomical considerations. Many patients with category-related disorders affecting living items have a diagnosis of herpes simplex encephalitis (HSE), associated with bilateral damage to the medial and inferior temporal lobes (Gainotti, 2000; Gainotti, Silveri, Daniele & Giustolisi, 1995). PET and imaging studies converge in suggesting occipital-temporal areas are associated with knowledge about animals (Damasio et al., 1996; Martin, Wiggs, Ungerleider & Haxby, 1996; Perani, et al., 1995; Tranel, Damasio, and Damasio, 1997). Defective knowledge about tools is associated with lateral temporo-parietal-occipital lesions (Tranel et al., 1997), and defective tool naming with posterolateral inferior temporal cortex and temporo-parietal junction damage (Damasio et al., 1996). PET studies (Damasio et al., 1996; Martin et al., 1996; Moore & Price, 1999; Perani et al., 1995) have reported activation in the left middle temporal gyrus and in the dorsolateral frontal cortex for artifacts. More critical evidence for modality-specific representations comes from recent work by Martin and colleagues (Beauchamp, Lee, Haxby & Martin, 2002; Chao, Haxby & Martin, 1999; Chao & Martin, 1999; Chao, Weisberg & Martin, 2002; Ishai, Ungerleider, Martin, Schouten & Haxby, 1999). Three main The Meanings of Objects and Actions 9 findings from this group strongly suggest a modality-specific featural organization. First, in fine-grained investigations of ventral temporal cortex they found that biological entities (faces and animals) were associated with greater activation in the lateral fusiform gyrus while activation for tools and houses was more medial. Crucially, activations for another non-living category (chairs) did not fall within the areas of maximal activation for tools and houses but was lateral (rather than medial) to the areas active for faces, falling in the inferior temporal gyrus. This finding argues against a strict living-nonliving separation and is, instead, compatible with different featural composition of the investigated categories. Second, within the lateral temporal cortex, specific activations were found for tool motion (left posterior medial temporal gyrus, an area just anterior to MT), while biological motion elicited greater activation in the superior temporal sulcus (Beauchamp, et al., 2002). These specific activations were strictly associated to the type of motion of the entities not to their belonging to different categories. Finally, all these studies showed that category-related responses are not restricted to a single region which responds maximally for that category, but that all categories activated a broad and largely overlapping region of the ventral and lateral temporal cortex and that the profile of activation differed depending on category. The latter suggests that object concepts are represented according to object features, rather than according to semantic categories corresponding to specific and anatomically segregated modules. Such a conclusion depends on showing that concepts in different domains of knowledge differ in their featural composition. In order to evaluate differences in featural composition, following Cree and McRae (2003) and McRae and Cree (2002), we developed featural norms, both for concepts referring to objects and concepts referring to actions. Given these norms, we then assess the relevance of different types of features in different semantic fields. Featural Norms Data for conceptual featural representations was gathered by asking speakers to generate features sufficient to define and describe different words, following McRae et al. (1997). In contrast to other tools to investigate semantic representations such as semantic differential scaling (Osgood, Suci & Tannenbaum, 1957); item similarity judgments (e.g. Romney, Moore & Rusch, 1997) or feature strength rating tasks (Schlesinger, 1995), they are less strongly experimenter-driven. In contrast to other experimenter-independent tools such as similarity measures based on text analysis (Landauer & Dumais, The Meanings of Objects and Actions 1